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ABSTRACT:  This  paper  presents  automated  methods  for  facilitating  after  action  review  in  team  training  exercises.  Much  of 
the  learning  from  team  training  arises  from  frank  after-the-fact  discussions  of  the  exercise,  combining  individual  attributions 
of  blame  or  credit  into  a  more  objective  view  of  what  transpired.  These  individual  attributions  are  social  judgments  involv¬ 
ing  not  only  causality  but  also  explanations  of  individual  responsibility,  free  will  and  mitigating  circumstances.  Such  judg¬ 
ments  are  a  key  aspect  of  social  intelligence  and  underlie  social  planning,  social  learning,  natural  language  pragmatics  and 
computational  models  of  emotion.  Here  we  introduce  a  computational  model  of  this  judgment  process  based  on  psychologi¬ 
cal  Attribution  Theory  and  discuss  its  potential  to  facilitate  after  action  review  in  team  training. 


1  Introduction 

This  paper  addresses  an  important  issue  in  team  training: 
the  problem  of  giving  credit  where  credit  is  due  when 
multiple  parties  are  involved  in  a  training  exercise.  Many 
training  applications  such  as  military  training  or  urban 
disaster  simulations  involve  distributed  decision-making 
and  distributed  knowledge.  No  one  individual  has  com¬ 
plete  authority  or  complete  access  to  information.  The 
shared  nature  of  group  training  considerably  complicates 
the  problem  of  generating  feedback  to  individuals  on 
where  they  are  performing  well,  where  they  need  im¬ 
provement,  and  whether  perceived  failures  are  actually 
outside  the  individual’s  control.  In  this  article  we  intro¬ 


duce  automated  techniques  that  facilitate  credit  assign¬ 
ment  in  team  training  simulations. 

Team  training  simulations  provide  individual  procedural 
training  during  the  course  of  the  exercise,  but  much  of  the 
learning  occurs  after  the  fact,  through  the  process  of  after 
action  review  (AAR).  This  is  an  after-the-fact  discussion 
of  an  exercise  where  participants  can  discuss  events  from 
multiple  perspectives  and  come  to  an  individual  under¬ 
standing  of  what  happened,  why,  and  how  to  sustain 
strengths  and  improve  weaknesses.  Bring  together  con¬ 
trasting  explanations  is  often  the  key  element  in  AAR.  In  a 
team  exercise,  individuals  view  success  and  failure  through 
the  lens  of  their  individual  perceptions  and  decisions,  but 
given  the  distributed  nature  of  these  exercises,  an  individ- 
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Figure  1 :  Interaction  between  the  trainee  and  autonomous 
characters  in  the  Mission  Rehearsal  Exercise 


makers  (e.g.  the  autonomous  agents)  are  incapable  of 
participating  in  the  AAR  process. 

This  paper  lies  out  a  preliminary  computational  approach 
to  forming  social  explanations  based  on  psychological 
attribution  theory.  We  see  several  immediate  applications 
of  this  model.  Here  we  focus  on  its  potential  to  facilitate 
the  after  action  review  process.  We  see  two  complemen¬ 
tary  vectors  to  apply  this  technology  to  AAR.  First,  these 
techniques  can  assist  an  AAR  facilitator  by  analyzing 
features  of  a  simulation  and  communication  between  in¬ 
dividual  decision-makers,  and  identifying  key  decision 
events  where  a  different  decision  or  better  information 
may  have  led  to  a  better  outcome.  Second,  by  incorporat¬ 
ing  social  credit  attributions  into  autonomous  agents, 
these  agents  can  form  social  judgments  from  their  own 
perspective  and  potentially  participate  in  the  AAR  proc¬ 
ess,  for  instance,  by  answering  questions  on  whom  they 
blame  and  why. 


ual’s  perspective  is  frequently  biased  or  misleading.  For 
example,  what  seems  like  a  bad  decision  from  the  perspec¬ 
tive  of  a  subordinate  might  make  more  sense  if  he  were 
placed  in  his  commander’s  shoes.  By  collectively  discuss¬ 
ing  events  after  the  fact,  decision  makers  can  come  to  a 
truer  understanding  of  who  did  what,  why,  and  how  they 
can  do  better  the  next  time. 

Underlying  AAR  is  the  problem  of  forming  social  explana¬ 
tions.  In  reflecting  on  an  exercise,  individuals  must  form 
judgments  not  only  of  causality  but  individual  responsibil¬ 
ity,  free  will  and  mitigating  circumstances.  Did  an  indi¬ 
vidual  choice  cause  a  significant  outcome?  Were  they 
simply  following  orders?  Did  they  agree  with  the  orders 
and,  if  not,  was  this  disagreement  communicated  to  their 
superiors?  By  eliciting  such  social  explanations,  individu¬ 
als  can  move  beyond  their  gut  feelings  about  the  exercise 
and  contrast  and  compare  the  factors  underlying  these  at¬ 
tributions. 

Facilitating  an  effective  AAR  is  an  art  and  it  is  particu¬ 
larly  difficult  in  large  training  simulations.  Effective 
AAR  demands  an  expert  facilitator  that  has  a  sense  of  the 
key  events  in  the  exercise  and  can  guide  and  focus  the 
discussion.  It  also  demands  that  participants  can  reflect 
on  their  individual  attributions  of  blame  and  credit  and 
relate  the  underlying  factors  that  led  to  them.  Neither 
demand  is  easily  met.  Eor  example  in  military  exercises 
such  as  the  Simulated  Theater  of  War  (STOW),  thou¬ 
sands  of  entities  involved.  Decisions  are  being  made  at 
multiple  levels  in  the  command  hierarchy,  and  many  de¬ 
cision-makers  are  autonomous  or  semi-autonomous 
agents  with  opaque  reasoning  processes.  In  such  exer¬ 
cises,  facilitators  can  be  easily  overwhelmed  by  the  com¬ 
plexity  of  the  exercise  and  many  of  the  key  decision¬ 


2  Motivating  Example 

We  have  been  developing  these  techniques  in  the  context 
of  an  Army  leadership-training  simulator  [Rickel  et  al., 
2002],  and  an  example  from  this  system  illustrates  the  so¬ 
cial  factors  involved  in  judgments  of  blame  and  credit.  The 
Mission  Rehearsal  Exercise  (MRE)  is  a  virtual  reality 
training  environment  designed  to  teach  decision-making 
skills  in  high-stakes  social  situations.  Intelligent  agents 
control  characters  (virtual  humans)  in  the  virtual  environ¬ 
ment,  playing  the  roles  of  locals,  friendly  and  hostile 
forces,  and  other  mission  team  members.  The  goal  is  to 
support  realistic  face-to-face  interactions,  requiring  an  em¬ 
phasis  on  creating  “broad  agents”  that  integrate  motor 
skills,  problem  solving,  emotion,  gestures,  facial  expres¬ 
sions,  and  language.  The  virtual  humans  engage  in  task- 
oriented  reasoning,  and  communicate  through  verbal  and 
non-verbal  behavior,  including  emotional  responses.  The 
goal  is  to  support  training  the  social  and  human-centered 
aspects  of  command  decision  making. 

Although  the  MRE  has  currently  focused  on  small  unit 
operations,  it  embodies  essential  features  of  many  group 
decision-making  problems,  including  the  fact  that  author¬ 
ity,  decision-making  responsibility,  and  perception  are 
distributed  across  a  group  of  individuals.  In  the  simula¬ 
tion,  the  trainee  is  placed  in  command  of  an  infantry  pla¬ 
toon,  Eagle  2-6,  supporting  peacekeeping  operations  near 
the  Bosnian  city  of  Celic.  The  trainee’s  mission  is  to  rein¬ 
force  another  unit.  Eagle  1-6.  In  route,  one  of  his  vehi¬ 
cles  seriously  injures  a  civilian,  and  the  international 
press  is  already  on  scene.  The  trainee  must  balance 
whether  to  continue  the  mission  or  render  aid.  Many  de¬ 
cisions  and  outcomes  are  possible.  In  the  following  ex¬ 
ample,  the  trainee  decides  to  split  his  forces,  ordering  his 
sergeant  to  send  half  of  his  squads  to  aid  the  other  unit. 
His  sergeant  responds  that  this  is  a  bad  idea;  it  will  alio- 
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Figure  2:  Shaver’s  Attribution  Model  of  Blame,  simplified  and  adapted  from  [Shaver  1985], 


cate  insufficient  forces  to  either  goal,  and  instead,  one 
squad  should  be  sent  ahead  to  scout  the  route.  The  trainee 
overrules  this  recommendation,  restating  the  original  or¬ 
der.  In  the  end,  the  trainee  finds  he  has  insufficient  re¬ 
sources  to  render  aid  in  a  timely  manner,  a  fact  that  is 
duly  noted  in  the  nightly  news.  The  central  question  ad¬ 
dressed  here  is  to  assess  who,  if  anyone  deserves  blame 
for  this  unfortunate  outcome.  Did  the  trainee  truly  have  a 
more  effective  alternative?  Did  he  have  sufficient  infor¬ 
mation  to  make  an  effective  decision?  Was  the  problem 
with  the  order  or  how  it  was  carried  out? 

Individuals  may  differ  in  whom  they  praise  or  blame  in  a 
specific  situation,  but  psychologists  agree  on  the  broad 
features  individuals  use  to  make  such  judgments.  Did  the 
agent  cause  the  outcome?  Did  he/she  intend  the  act?  Did 
he/she  know  the  consequence?  Did  he/she  have  choice  or 
was  he/she  coerced  by  another  agent?  In  the  example,  we 
may  infer  from  the  conversation  that  there  were  alterna¬ 
tives  and  the  sergeant  was  coerced  by  the  trainee  to  fol¬ 
low  an  undesirable  course  of  action.  We  can  further  sur¬ 
mise  that  the  trainee  must  have  known  the  consequence 
since  his  sergeant  said  so.  Baring  unknown  mitigating 
factors  (e.g.  the  sergeant  always  gave  bad  advice  in  the 
past),  we  would  likely  conclude  that  the  trainee  is  to 
blame  for  the  delay  and  negative  press.  This  example 
shows  that  proper  assignment  of  credit  or  blame  in  a  so¬ 
cial  setting  must  not  only  consider  the  actions  (both 
physical  acts  and  speech  acts)  and  knowledge  state  of 
different  actors,  but  also  need  to  make  use  of  information 
available  to  reason  about  key  attributions  that  contribute 
to  the  judgment  process. 


3  Attribution  Theory  for  Social  Judgment 

The  assignment  of  social  credit  or  blame  has  been  studied 
extensively  in  philosophy,  law,  and  social  psychology.  As 
our  primary  goal  is  to  inform  the  design  of  training  simula¬ 
tions  that  model  human  behavior  [Gratch  et  al,  2002],  our 
models  focus  on  descriptive  rather  than  proscriptive  mod¬ 
els  (i.e.,  what  people  do  rather  than  what  they  should  do). 
In  contrast,  much  of  the  related  work  on  AI  has  focused  on 
trying  to  identify  "ideal"  principles  of  responsibility  (e.g. 
the  legal  code  or  philosophical  principles)  and  ideal 
mechanisms  to  reason  about  these,  typically  contradictory 
principles  (e.g.,  non-monotonic  or  case-based  reasoning) 
[McCarty,  1997]. 

In  modeling  the  assignment  of  social  credit,  we  build  on 
psychological  attribution  theory,  specifically  the  work  of 
Weiner  [1995]  and  Shaver  [1985]  as  they  are  readily 
adapted  to  artificial  intelligence  knowledge  representa¬ 
tion  and  reasoning  methods.  Shaver’s  model  is  illustrated 
in  Figure  2.  In  these  models,  the  assignment  of 
credit/blame  is  a  multi-step  process  that  is  initiated  by 
events  with  positive  or  negative  consequences.  First  one 
assesses  causality,  distinguishing  between  personal  ver¬ 
sus  impersonal  causality  (i.e.  whether  the  consequence  is 
caused  by  a  human  or  by  environmental  factors).  If 
caused  by  a  human,  judgment  proceeds  by  assessing  key 
factors:  was  it  the  actor’s  intention  to  produce  the  out¬ 
come;  did  the  actor  foresee  its  occurrence;  was  the  actor 
forced  under  coercion  (e.g.,  was  the  actor  acting  under 
orders)?  As  the  last  step  of  the  process,  proper  degree  of 
credit/blame  is  assigned  to  the  responsible  agent.  Causal- 


ity,  intention  and  foreseeability  map  to  standard  concepts 
in  agent-based  systems,  particularly  frameworks  that  ex¬ 
plicitly  represent  beliefs,  desires  and  intentions  [Bratman, 
1987;  Pollack,  1990;  Grosz  and  Kraus,  1996].  Coercion 
requires  some  representation  of  social  relationships  and 
some  understanding  of  the  extent  to  which  the  coercion 
limits  one’s  range  of  options.  For  example,  one  may  be 
ordered  to  carry  out  a  task  but  to  satisfy  the  order,  there 
may  be  many  alternative  ways  that  vary  blame  or  credit- 
worthiness. 

A  system  to  support  the  AAR  process  based  on  attribu¬ 
tion  theory  must  be  able  to  rapidly  scan  a  trace  of  simula¬ 
tion  events  to  assess  instances  of  causality,  intentionality, 
foreknowledge  and  coercion.  In  a  standard  AAR  process, 
this  knowledge  comes  from  asking  people  to  elicit  their 
mental  state  at  the  time  of  certain  simulation  events. 
When  a  simulation  involves  computer-generated  decision 
makers,  it  may  be  possible  to  encode  such  information 
directly  in  the  simulation  trace  [Johnson,  1994].  In  gen¬ 
eral,  however,  we  would  like  to  model  the  process  that 
people  generally  use  in  attributing  credit/blame,  rather 
than  assume  the  perceiving  agent  has  privileged  access  to 
other’s  mental  state.  In  human  social  interactions,  such 
attributions  are  gleaned  from  a  variety  of  sources:  from 
observation  of  behavior,  from  statements  made  through 
natural  language,  from  knowledge  and  models  built  up 
through  past  interactions,  stereotypes,  and  cultural  norms. 
We  show  how  to  infer  such  information  by  analyzing 
communication  traces  between  entities  and  by  making 
use  of  agents’  knowledge  of  actions  and  consequences 
and  commonsense  intuition. 

3.1  Representation 

Automated  techniques  to  inform  social  judgments  must 
have  some  representation  of  the  team  exercise  including 
task  knowledge  (e.g.,  tasks,  their  effects,  alternative 
courses  of  actions),  state  knowledge  (e.g.,  information 
from  sensors  and  communication  events)  and  social 
knowledge  (e.g.,  power  relationships  between  individuals 
and  social  obligations).  We  build  on  standard  representa¬ 
tions  that  have  been  developed  by  planning  and  communi¬ 
cation  researchers. 

Tasks  consist  of  a  set  of  steps,  each  of  which  is  either  a 
primitive  action  (e.g.,  a  physical  or  sensing  action  in  the 
virtual  world)  or  an  abstract  action.  Abstract  actions  may 
be  decomposed  hierarchically  in  multiple  ways  and  each 
decomposition  consists  of  a  sequence  of  abstract  or 
primitive  sub-actions.  Interdependencies  among  steps  are 
represented  as  a  set  of  causal  links  and  threat  relations. 
Each  causal  link  specifies  that  an  effect  of  a  step  achieves 
a  particular  goal  that  is  a  precondition  for  another  step. 
For  example,  marking  a  landing  zone  with  smoke 
achieves  the  goal  of  visually  identifying  the  landing  zone 
for  the  helicopter,  which  is  a  precondition  for  landing  it. 


Threat  relations  specify  that  an  effect  of  a  step  threatens  a 
causal  link  by  making  the  goal  unachievable  before  it  is 
needed.  For  example,  extinguishing  the  smoke  before  the 
helicopter  arrives  threatens  its  ability  to  land. 

In  addition  to  understanding  the  structure  of  tasks,  agents 
must  understand  the  roles  of  each  team  member.  Each 
task  step  is  associated  with  the  team  member  that  per¬ 
forms  it,  the  performer.  In  addition,  each  task  step  may 
be  annotated  with  the  teammate  who  has  authority  over 
its  execution:  the  performer  of  a  task  step  cannot  execute 
it  until  authorization  is  given  by  the  specified  teammate 
with  authority.  This  requires  modeling  hierarchical  or¬ 
ganizational  structure  of  social  teams,  such  as  in  the  mili¬ 
tary. 

Finally,  we  must  model  the  impact  of  communication 
events  between  entities  in  the  simulation.  We  follow  the 
Trindi  project  approach  to  communication  management 
[Farsson  and  Traum,  2000].  This  approach  maintains  an 
explicit  information  state  that  is  updated  by  dialogue 
moves.  For  example,  an  assertion  by  a  speaker  is  a  dia¬ 
logue  move  that  has  the  effect  of  establishing  a  commit¬ 
ment  by  the  speaker  that  the  assertion  is  true.  Orders,  on 
the  other  hand,  can  only  be  issued  by  a  superior  to  a  sub¬ 
ordinate  in  the  social  structure  and  have  as  their  effect  a 
social  obligation  for  the  subordinate  to  perform  the  con¬ 
tent  of  the  action. 

3.2  From  theory  to  computational  approach 

Given  the  representation,  we  can  now  revisit  the  question 
of  representing  the  core  conceptual  variables  underlying 
Shaver  and  Weiner’s  attribution  theories. 

Causality:  Causal  knowledge  is  encoded  via  hierarchical 
task  representation.  Each  action  consists  of  a  set  of  pre¬ 
conditions  and  effects.  The  desirability  of  action  effects 
(i.e.  effects  having  positive/negative  significance  to  an 
agent)  is  represented  by  utility  values  [Blythe,  1999]. 

A  non-decision  node  is  an  abstract  action  that  can  only  be 
decomposed  in  one  way.  A  decision  node,  on  the  other 
hand,  can  be  decomposed  in  more  than  one  way.  In  a 
decision  node,  an  agent  needs  to  make  a  decision  and 
select  among  different  choices.  If  a  decision  node  A  can 
be  decomposed  in  different  ways  aj,  02,  ...,  a„,  we  call  O], 
02,  ...,  a„  choices  of  action  A,  and  a 2,  02,  ...,  a„  are  called 
alternatives  of  each  other.  Clearly,  a  primitive  action  is  a 
non-decision  node,  while  an  abstract  action  can  be  either 
a  non-decision  node  or  a  decision  node. 

Consequences  or  outcomes  (we  use  these  two  terms 
equally  in  this  paper)  of  actions  are  represented  as  a  set  of 
primitive  action  effects.  The  consequence  set  of  an  action 
A  can  be  defined  recursively  from  leaf  nodes  (i.e.  primi¬ 
tive  actions)  in  plan  structure  to  action  A  as  follows.  Con- 


sequences  of  a  primitive  action  are  those  effects  with 
non-zero  utility.  For  an  abstract  action,  if  the  abstract 
action  is  a  non-decision  node,  then  the  consequence  set  of 
the  abstract  action  is  the  aggregation  of  the  consequences 
of  its  sub-actions.  If  the  abstract  action  is  a  decision  node, 
we  need  to  differentiate  two  kinds  of  consequences.  If 
consequence  p  occurs  among  all  the  choices  of  a  decision 
node,  we  call  p  a  common  consequence  of  the  decision 
node;  otherwise  p  is  a  non-common  consequence  of  the 
decision  node.  Consequence  set  of  a  decision  node  is 
defined  as  the  set  of  its  common  consequences. 

Foreseeability  refers  to  an  agent’s  foreknowledge  about 
actions  and  consequences.  We  use  know  and  bring-about 
to  express  foreseeability.  If  an  agent  knows  an  action  A 
brings  about  a  consequence  p  before  the  execution  of  A, 
then  the  agent  foresees  A  brings  about  p. 

Intentionality:  Intention  is  generally  conceived  as  a 
commitment  to  work  toward  a  certain  act  or  outcome.  A 
key  question  of  credit  assignment  is  distinguishing 
whether  an  entity  intends  an  act  (act  intent)  versus 
whether  it  intends  specific  consequences  of  the  action 
(outcome  intent).  Most  theories  argue  that  outcome  intent 
rather  than  act  intent  is  the  key  factor  in  determining 
credit/blame.  This  is  illustrated  by  the  package  deal  prob¬ 
lem  [Bratman,  1990].  Say  military  planners  wish  to  bomb 
a  weapons  factory  but  a  school  is  placed  within  the  fac¬ 
tory.  One  might  assume  the  planners  intend  the  act  of 
bombing  and  intend  the  outcome  of  destroying  the  fac¬ 
tory  but  not  intend  the  outcome  of  destroying  the  school. 

Following  the  notations  in  [Grosz  and  Kraus,  1996],  we 
use  intend-to  and  intend-that  to  distinguish  act  intention 
and  outcome  intention.  But  since  our  work  is  applied  to 
richer  social  context,  we  extend  the  meaning  of  intend-to 
to  include  indirect  cases.  One  case  is  that  an  agent  may 
intend  to  act,  but  may  not  be  the  actor  himself/herself 
(e.g.  by  ordering  another  agent  to  act).  Another  case  is 
that  an  agent  may  intend  to  act  but  is  coerced  to  do  so. 
Intend-that,  on  the  contrary,  is  used  in  a  more  restricted 
manner.  Because  of  the  nature  of  our  problem,  we  always 
specify  intending  an  outcome  of  which  action. 

Coercion;  Similar  difference  exists  in  act  coercion  and 
outcome  coercion.  An  agent  may  be  coerced  to  act  yet 
not  be  necessarily  coerced  to  achieve  any  specific  out¬ 
come  assuming  if  there  are  multiple  ways  to  achieve  the 
task.  It  is  important  to  differentiate  being  coerced  to  act 
and  being  coerced  to  achieve  consequence(s)  of  the  ac¬ 
tion,  because  it  is  the  latter  that  actually  influence  our 
judgment  of  behavior,  and  is  used  to  determine  praise¬ 
worthy/blameworthy  agent.  We  use  coerced-to  and  co- 
erced-that  to  distinguish  coerced  actions  and  coerced 
consequences.  In  the  case  of  outcome  coercion  (i.e.  co- 
erced-that  is  true),  the  responsible  agent  for  a  conse¬ 


quence  is  the  performer  of  an  action  or  the  entity  that  has 
authority  over  the  action,  but  the  action  may  not  be  the 
primitive  one  that  directly  leads  to  the  outcome. 

3.3  The  Attribution  Process 

Social  credit  attributions  are  always  from  a  perceiver’s 
perspective.  For  the  purposes  of  AAR,  this  could  be  a 
global  perspective  with  access  to  all  simulation  events  and 
communications  if  the  goal  is  to  inform  an  exercise  con¬ 
troller.  Or  it  could  be  from  the  perspective  of  a  key  deci¬ 
sion  maker  if  the  goal  is  to  allow  an  autonomous  agent  to 
answer  questions  during  the  AAR  process.  Since  different 
perceivers  may  have  different  goals,  different  observations, 
and  different  knowledge  about  the  world,  it  may  well  be 
the  case  that  for  the  same  situation,  different  perceivers 
form  different  judgments.  For  example,  an  agent  may  not 
think  himself/herself  is  blameworthy,  but  the  perceiver 
thinks  the  agent  is. 

Nevertheless,  the  attribution  process  is  general,  and  ap¬ 
plied  uniformly  to  different  perceivers.  If  an  action  per¬ 
formed  by  an  agent  brings  about  positive/negative  conse¬ 
quence,  and  the  agent  is  not  coerced  to  achieve  the  con¬ 
sequence,  then  credit/blame  is  assigned  to  the  performer 
of  the  action.  Otherwise,  assign  credit/blame  to  the  au¬ 
thority.  If  the  authority  is  in  turn  coerced,  the  process 
needs  to  trace  further  up  the  hierarchy  to  find  the  respon¬ 
sible  agent  for  the  consequence. 

Coercion  is  used  to  determine  the  praisewor¬ 
thy/blameworthy  agent,  while  intention  and  foreseeability 
are  used  in  assigning  the  degree  of  praiseworthiness/ 
blameworthiness.  We  use  a  simple  categorical  model  of 
intensity  assignment,  though  one  could  readily  extend  the 
model  to  a  numeric  intensity  value  by  incorporating 
probabilistic  rules  of  inference.  If  the  responsible  agent 
intends  a  consequence  while  acting,  the  intensity  assigned 
is  high.  If  the  responsible  agent  does  not  foresee  the  con¬ 
sequence,  the  intensity  is  low. 

4  Inference  from  Communication  and  Plans 

Judgments  of  causality,  forseeablity,  intentionality  and 
coercion  are  informed  by  evidence  extracted  from  commu¬ 
nication  events  and  from  task  knowledge.  We  have  devel¬ 
oped  a  number  of  commonsense  rules  that  allow  an  auto¬ 
mated  system  to  make  inferences  based  on  this  evidence. 

4.1  Inferring  from  Communication  Events 

Group  simulations  often  explicitly  represent  communica¬ 
tion  events  between  entities  and  considerable  attribution¬ 
relevant  information  can  be  extracted  from  this  communi¬ 
cation.  For  example,  languages  like  CCSIL  [Salisbury, 
1995]  or  KQML  [Finin  et  al.,  1994]  represent  communica¬ 
tion  between  entities  in  terms  of  abstract  speech  acts  [Aus¬ 
tin,  1962;  Searle,  1969].  When  a  simulation  involves  natu¬ 
ral  language  communication  between  human  participants. 


underlying  speech  acts  can  be  acquire  using  natural  lan¬ 
guage  processing  technology. 

We  assume  communication  between  agents  is  grounded 
[Traum,  1994]  and  agents  communicate  sincerely  and 
relevantly  in  conversation  [Grice’s  maxims,  1975].  Back¬ 
ground  information  (e.g.  agents’  social  roles,  relationship, 
etc)  is  also  important,  for  example,  an  order  can  be  suc¬ 
cessfully  issued  only  to  subordinates;  but  a  request  can  be 
made  of  any  agent. 

When  a  speech  act  is  performed,  a  perceiving  agent  ob¬ 
serves  the  conversation  and  makes  inferences  based  on 
his/her  beliefs.  As  the  conversation  proceeds,  the  per- 
ceiver  acquires  new  beliefs  and  updates  inferences  ac¬ 
cordingly. 

For  our  purpose,  we  are  interested  in  analyzing  speech 
acts  that  help  infer  agents’  desires,  intentions,  foreknowl¬ 
edge  and  choices  in  acting.  We  consider  the  following 
speech  acts  (x  and  y  are  different  agents.  A  and  B  are  ac¬ 
tions.  p  is  a  proposition  and  f  is  a  time  stamp): 

infomi(A',  y,  p,  t):x  informs  y  that  p  at  time  t. 
order(j\:,  y,  A,  t):x  orders  y  to  do  A  at  time  t. 
request(x,  y,  A,  t):x  requests  y  to  do  A  at  time  t. 
accept(x.  A,  t):  x  accepts  to  do  A  at  time  t. 
reject(j\:,  A,  t):  x  rejects  to  do  A  at  time  t. 

counterpropose(j\:,  A,  B,  y,  t):  x  counters  A  and  proposes  B  to  y  at  time  t. 

We  have  identified  several  commonsense  inference  rules 
that  allow  perceiving  agents  to  form  inferences  from 
communication  patterns.  These  rules  are  general,  so  can 
be  combined  flexibly  and  applied  to  variable  communica¬ 
tion  sequences  of  multiple  participants.  Here  we  illustrate 
two  to  give  a  flavor  of  the  approach. 

An  order  or  a  request  gives  evidence  that  the  speaker 
desires  the  listener  to  act: 

order(y,  z,  A,  tl)  A  tl<t2  A  -i(3t3)(tl<t3<t2  a  believe(x,  -iwant(y, 
do(z,  A)),  t3))  =>  believe(x,  want(y,  do(z,  A)),  t2) 
request(y,  z,  A,  tl)  A  tl<t2  A  -i(3t3)(tl<t3<t2  a  believe(x,  -iwant(y, 
do(z,  A)),  t3))  =>  believe(x,  want(y,  do(z,  A)),  t2) 

The  listener  may  accept,  reject  or  counter-propose  an 
order/request.  Various  inferences  can  be  made  depending 
on  the  response  and  the  power  relationship  between  the 
speaker  and  the  listener.  For  instance,  if  the  listener  ac¬ 
cepts  an  act  wanted  by  a  superior,  there  is  evidence  of 
coercion,  and  the  speaker  is  viewed  as  the  coercing  agent 
of  the  action. 

believe(x,  want(y,  do(z,  A)),  tl)  A  accept(z,  A,  t2)  A  tl<t2<t3  a 
-i(3t4)(t2<t4<t3  A  believe(x,  -iintend-to(z,  A),  t4))  =>  believe(x,  in- 
tend-to(z,  A),  t3) 

believe(x,  want(y,  do(z,  A)),  tl)  a  accept(z,  A,  t2)  A  superior(y,  z)  a 
tl<t2<t3  A  -i(3t4)(t2<t4<t3  a  believe(x,  -icoerced-to(z,  A),  t4))  =>  be- 
lieve(x,  coerced-to(z,  A),  t3) 


Because  being  coerced  to  act  implies  intending  to  act,  the 
two  forms  of  the  rule  are  consistent.  The  second  form  is 
more  specific  and  thus  overrides  the  first  one  if  both  are 
activated. 

4.2  Inferring  from  Plans 

Conversational  dialogs  and  Speech  acts  provide  informa¬ 
tion  about  agents’  intentions  and  choices  in  acting,  i.e.  in- 
tend-to  and  coerced-to,  but  the  attribution  process  actually 
uses  intend-that  and  coerced-that  for  judgment.  So  we 
need  to  solve  the  problem  of  inferring  outcome  intention 
and  outcome  coercion  from  act  intention  and  act  coercion. 

Different  agent  may  have  access  to  different  plans  in 
memory.  While  plans  are  specific  to  certain  domain,  the 
structure  of  plans  can  be  described  using  domain- 
independent  terms  such  as  action  types,  alternatives  and 
action  effects.  To  solve  the  problem  in  a  general  way,  we 
make  use  of  the  hierarchical  task  structures,  by  differenti¬ 
ating  action  types,  comparing  consequences  of  alterna¬ 
tives,  and  separating  common  consequences  of  an  action 
from  its  non-common  ones. 

Inferring  Outcome  Intents  from  Act  Intents 

A  key  question  in  assigning  blame  is  distinguishing 
whether  an  entity  intends  an  act  (act  intent)  and  whether 
the  entity  intends  certain  consequences  and  side  effects  of 
that  action  (outcome  intent).  A  number  of  rules  of  evi¬ 
dence  can  assess  these  distinctions  by  considering  the  task 
structure.  For  example,  if  an  entity  intends  to  perform  an 
act,  the  entity  must  intend  to  achieve  (at  least)  one  conse¬ 
quence  of  the  action.  If  the  action  has  only  one  conse¬ 
quence,  then  the  entity  must  intend  that  consequence.  In 
more  general  cases,  when  an  action  has  multiple  conse¬ 
quences,  to  infer  an  entity’s  intention  in  achieving  a  par¬ 
ticular  outcome,  a  perceiver  may  examine  alternative  acts 
the  agent  intends  and  does  not  intend,  and  compare  the 
consequences  of  intended  and  unintended  alternatives. 

We  have  developed  a  number  of  rules  of  evidence  to  dis¬ 
tinguish  intend-to  from  intend-that.  For  example,  con¬ 
sider  the  case  illustrated  in  Figure  4  where  there  are  two 
alternatives  ways  to  decompose  an  act  (one-squad- 
forward  or  two-squads-forward).  From  the  sergeant’s 
perspective,  these  alternatives  have  similar  consequences 
with  the  exception  that  two  squads-forward  has  an  addi¬ 
tional  consequence  that  the  sergeant  considers  bad.  If  in 
particular,  there  is  only  one  non-shared  consequence  p  of 
A  that  does  not  occur  in  the  consequence  set  of  B,  the 
agent  must  intend  p. 

believe(x,  intend-to(y,  A))  a  believe(x,  — icoerced-to(y,  A))  a  believe(x, 
— iintend-to(y,  B))  a  believe(x,  altemative(A,  B))  a  believe(x,  conse- 
quence(B)cconsequence(A))  =>  3p(believe(x,  pG  consequence(A))  a 
believe(x,  p£  consequence(B))  a  believe(x,  intend-that(y,  p.  A))) 


As  one  makes  such  inference,  solutions  to  inferring  out¬ 
come  intents  are  partial  depending  on  the  information 
available  and  comparative  features  of  consequence  sets  of 
alternatives. 

Inferring  Outcome  Coercion  from  Act  Coercion 

In  a  non-decision  node,  if  an  agent  is  coerced  to  act,  the 
agent  is  also  coerced  to  achieve  the  consequence  of  subse¬ 
quent  actions,  for  the  agent  has  no  other  choice. 

believe(x,  coerced-to(y,  A))  a  believe(x,  non-decision-node(A))  a  be- 

lieve(x,  pG  consequence(A))  =>  believe(x,  coerced-that(y,  p,  A)) 

In  a  decision  node,  however,  an  agent  must  make  deci¬ 
sion  amongst  multiple  choices.  Even  if  an  agent  is  co¬ 
erced  to  act,  it  does  not  follow  that  the  agent  is  coerced  to 
achieve  certain  consequence  of  the  subsequent  actions.  In 
order  to  infer  coerced-that  from  coerced-to  in  a  decision 
node,  we  can  examine  the  choices  of  the  decision  node.  If 
an  outcome  is  a  common  consequence  of  the  node,  then  it 
is  unavoidable:  coerced-that  is  true.  Otherwise,  if  an  out¬ 
come  is  a  non-common  consequence  of  the  node,  which 
means  the  agent  has  option  to  choose  the  alternative  that 
avoids  this  outcome,  then  coerced-that  is  false.  Our  defi¬ 
nition  of  consequence  set  ensures  the  consistency  when 
the  rules  are  applied  to  the  nodes  of  different  levels  in 
plan  structure. 

Back-Tracing  Algorithm 

Judgment  of  attributions  is  made  after  the  fact  (i.e.  actions 
have  been  executed  and  the  consequence  has  occurred). 
The  evaluation  process  starts  from  the  primitive  action  that 
directly  causes  a  consequence  with  positive  or  negative 
utility.  Since  coercion  may  occur  in  more  than  one  level  in 
hierarchical  plan  structure,  the  process  must  trace  from  the 
primitive  action  to  the  higher-level  actions.  We  use  a  back- 
tracing  algorithm  to  determine  the  responsible  agent.  The 
algorithm  takes  as  input  some  desirable  or  undesirable 
consequence  of  a  primitive  action  and  works  up  the  task 
hierarchy.  During  each  pass  through  the  main  loop  the 
algorithm  uses  dialogue  and  plan  evidence  rules  to  assign 


Algorithm  {consequence,  plan): 

1.  parent=A,  where  effect  of  action  A  is  consequence 

2.  DO 

2.1  node=parent 

2.2  coerced-toiperformerinode),  node)=unknown 
coerced-that{performer{node), consequence,  node)=unknown 
responsible{consequence)=performer{node) 

2.3  Search  dialog  history  on  node  and  apply  dialog  rules 

2.4  IF  coerced-toiperformerinode),  node) 

2.5  THEN  apply  inference  rules  on  node 

2.6  IF  coerced-that{performer{node),  consequence,  node) 

2.7  THEN  responsible{consequence)=authority{node) 

2.8  parent=V,  where  P  is  parent  of  node  in  plan 

WHIL£  parent^root  of  plan  AND  coerced-that{performer{node), 
consequence,  node)  is  true 

3.  RETURN  responsible{consequence),  node 

Figure  3.  Back-Tracing  Algorithm 


attributions  at  the  current  level  (2.3).  If  there  is  evidence 
that  the  performer  was  coerced  to  act  (2.4),  rules  of  evi¬ 
dence  also  assess  outcome  coercion  (2.5).  If  there  is  out¬ 
come  coercion  (2.6),  the  authority  is  deemed  responsible 
(2.7).  If  current  action  is  not  the  root  node  in  plan  struc¬ 
ture  and  outcome  coercion  is  true,  the  algorithm  enters 
next  loop  and  evaluates  the  next  level  up  in  the  task  hier¬ 
archy. 

After  the  execution  of  the  algorithm,  the  responsible 
agent  for  the  outcome  is  determined.  The  algorithm  may 
also  acquire  values  of  act  intention  and  foreknowledge 
meanwhile.  Then  rules  for  inferring  outcome  intents  can 
be  applied  to  further  determine  the  responsible  agent’s 
intention  in  achieving  the  evaluated  consequence. 

5  Illustration 

We  illustrate  the  process  of  credit  assignment  through  the 
MRE  example  introduced  above.  We  focus  on  three  social 
actors,  the  lieutenant,  the  sergeant  and  the  squad  leader, 
who  work  as  a  team  in  task  performance.  The  lieutenant  is 
a  human  trainee  and  acts  as  authority  over  the  sergeant. 
The  squad  leader  acts  as  a  subordinate  of  the  sergeant. 
Communications  between  agents  are  represented  via 
speech  acts  and  a  conversation  history  as  in  the  MRE. 

Take  the  sergeant’s  perspective  as  an  example.  The  ser¬ 
geant  perceives  the  conversation  between  the  actors  and 
task  execution.  Conversation  history  includes  the  follow¬ 
ing  acts,  ordered  by  the  time  speakers  addressed  them.  It, 
sgt  and  sld  stand  for  the  lieutenant,  the  sergeant  and  the 
squad  leader.  tl<t2<...<t6. 

1.  order(lt,  sgt,  two-squads-fwd,  tl) 

2.  inform(sgt,  It,  bring-about(two-squads-fwd,  unit-fractured,  t2) 

3.  counter-propose(sgt,  It,  two-squads-fwd,  one-squad-fwd,  t3) 

4.  order(lt,  sgt,  two-squads-fwd,  t4) 

5.  accept(sgt,  two-squads-fwd,  t5) 

6.  order(sgt,  sld,  l*'-and-4‘'’-forward,  t6) 

Figure  4  illustrates  a  partial  plan  the  sergeant  has  access  to. 
One  squad  forward  and  two  squads  forward  are  two 
choices  of  action  support  Eagle  1-6.  One  squad  forward  is 
composed  of  two  primitive  actions,  4'^  squad  recon  (for¬ 
ward)  and  remaining  (squads)  forward.  Two  squads  for¬ 
ward  consists  of  F‘  and  4*  (squads)  forward  and  2”“^  and 
5'^^  (squads)  forward.  Effects  of  primitive  actions  are 
shown  under  the  graph.  Assume  two  effects  are  salient  to 
the  sergeant.  1-6  supported  is  a  desirable  team  goal.  The 
sergeant  assigns  negative  utility  to  unit  fractured  and  that 
this  consequence  serves  as  input  to  the  back-tracing  algo¬ 
rithm.  We  illustrate  how  to  find  the  blameworthy  agent 
given  the  sergeant’s  knowledge  and  observations  of  com¬ 
munication  events  and  task  execution. 

Loop  1:  The  algorithm  starts  from  primitive  action  F'-and- 
4*-fwd,  of  which  unit  fractured  is  an  effect.  The  sergeant 
perceived  the  action  was  executed  by  the  squad  leader. 


Figure  4:  Team  Plan  from  Sergeant’s  Perspective 


Step  2.2:  By  default,  coercd-to{sld,  r‘-and-4‘^-jwd)  is 
unknown,  coerced-that(sld,  unit-fractured,  r‘-and-4‘^- 
fwd)  is  unknown.  Assign  the  squad  leader  to  the  respon¬ 
sible  agent. 

Step  2.3:  Relevant  conversation  history  is  act  6.  Since  the 
sergeant  ordered  the  squad  leader  the  action,  through  in¬ 
ference  rules  the  algorithm  infers  that  the  sergeant  be¬ 
lieves  that  he  wants  the  squad  leader  to  act.  Since  the 
squad  leader  accepted  by  executing  the  action  and  the 
sergeant  is  the  superior,  the  algorithm  infers  that  the  ser¬ 
geant  believes  that  he  coerced  the  squad  leader  to  act. 

Step  2.4-2. 5:  Since  coerced-to  is  tme  and  the  primitive 
action  is  a  non-decision  node,  then  through  the  applica¬ 
tion  of  an  inference  rule,  the  sergeant  believes  he  coerced 
the  squad  leader  to  fracture  the  unit. 

Step  2.6-2.7:  Since  coerced-that  is  tme,  assign  the  coerc¬ 
ing  agent  to  the  responsible  agent.  The  sergeant  believes 
he  is  responsible  for  unit-fractured. 

Since  parent  node  is  not  the  root  in  plan  stmcture  and  co- 
erced-that  is  true,  the  algorithm  enters  next  loop.  We  leave 
the  executions  of  subsequent  loops  to  the  reader.  The  main 
results  are  given  below. 


Loop  2:  The  action  is  two-squads-fwd,  performed  by  the 
sergeant.  Relevant  conversation  history  is  sequence  1-5. 
A  variety  of  beliefs  can  be  inferred  from  commonsense 
rules  by  analyzing  the  task  structure  and  communication 
history. 

believe(sgt,  want(lt,  do(sgt,  two-squads-fwd))) 

believe(sgt,  know(lt,  bring-about(two-squads-fwd,  unit-fractured))) 

believe(sgt,  know(lt,  altemative(one-squad-fwd,  two-squads-fwd))) 

believe(sgt,  intend-to(lt,  two-squads-fwd)) 

believe(sgt,  — iintend-to(lt,  one-squad-fwd)) 

believe(sgt,  coerced-to(sgt,  two-squads-fwd)) 

believe(sgt,  coerced-that(sgt,  unit-fractured,  two-squads-fwd)) 

After  loop  2,  the  sergeant  believes  the  lieutenant  coerced 
him  to  fracture  the  unit.  So  the  lieutenant  is  responsible 
for  the  outcome. 

Loop  3:  The  action  is  support-Eagle  1-6,  performed  by  the 
lieutenant.  There  is  no  relevant  conversation  in  history. 
The  values  of  coerced-to,  coerced-that  and  responsible 
agent  are  default.  There  is  no  clear  evidence  of  coercion,  so 
the  sergeant  believes  that  the  lieutenant  is  the  responsible 
agent.  Parent  node  is  the  root  in  plan  stmcture.  The  algo¬ 
rithm  terminates. 

Now  the  sergeant  also  has  the  belief  that  the  student  in¬ 
tended  to  send  two  squads  forward  and  did  not  intend  to 
send  one  squad  forward  (acquired  in  loop  2).  Since  the 
consequence  set  of  one-squad-fwd  (i.e.  1  -6-supported)  is 


subset  of  the  consequence  set  of  two-squads-fwd  (i.e.  1-6- 
supported  and  unit-fractured),  through  inference  rules  the 
algorithm  determines  the  sergeant  believes  that  the  lieu¬ 
tenant  intended  unit-fractured.  So  the  lieutenant  is  to 
blame  with  high  intensity. 

6  Conclusions 

A  key  element  of  the  after  action  review  process  is  to  form 
social  explanations  of  blame  or  credit  and  to  allow  various 
individuals  to  compare  and  contrast  their  individual  expla¬ 
nations  in  order  to  arrive  at  a  more  objective  understanding 
of  the  group  exercise.  In  this  article  we  present  a  prelimi¬ 
nary  computational  approach  to  automate  this  social  credit 
assignment  process.  Two  obvious  applications  of  this 
model  for  AAR  are  in  allowing  synthetic  agents  to  partici¬ 
pate  in  the  AAR  process  (by  forming  and  relating  their 
own  social  attributions)  and  by  assisting  a  human  AAR 
facilitator  by  automatically  processing  a  global  simulation 
trace  and  identifying  key  events  or  decisions  that  need  dis¬ 
cussion. 

The  problem  of  social  credit  assignment  is  central  in  so¬ 
cial  psychology  and  social  cognition.  With  the  develop¬ 
ment  of  human-like  agent  systems,  it  is  increasingly  im¬ 
portant  for  computer-based  systems  to  model  this  human¬ 
centric  form  of  social  inference.  Our  work  attempts  to 
help  bridge  between  psychological  accounts  and  compu¬ 
tational  models  by  means  of  AI  methods.  Rather  than 
impose  arbitrary  rules  on  judgment  process,  our  work 
relies  on  commonsense  heuristics  of  human  inference 
from  communication  events  and  plans  as  knowledge 
states  of  agents.  Our  treatments  are  domain-independent, 
thus  can  be  used  as  a  general  approach  to  the  problem. 

In  the  future,  we  must  incorporate  probabilistic  reasoning 
to  deal  with  uncertainty  in  observations  and  judgment 
process.  For  modeling  more  complex  multi-agent  team¬ 
work,  we  need  to  consider  joint  responsibility  and  sharing 
responsibility  among  teammates  (the  current  model  as¬ 
sumes  one  agent  has  sole  responsibility).  The  inferences 
related  to  foreseeability  are  too  restrictive  and  need  to 
make  better  use  of  plan  knowledge,  specifically  consider¬ 
ing  how  actions  may  be  coerced  not  just  directly  through 
orders,  but  indirectly  due  to  the  effects  of  actions  of  other 
entities.  For  example,  an  action  taken  by  one  entity  may, 
with  foreknowledge,  coerce  another  entity  to  pursue  a 
less  desirable  option.  As  our  task  representations  already 
encode  information  about  action  preconditions  and  ef¬ 
fects,  this  type  of  inference  is  a  natural  extension  of  our 
existing  methods. 
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